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PROJECT  SUMMARY 


The  primary  goal  of  this  research,  is  to  understand  how  human  observers  process 
temporally  patterned,  acoustic  stimuli.  We  have  been  performing  a  series  of  studies  of 
how  listeners  detect,  encode,  store,  and  compare  sequences  of  tones.  Several  studies  of 
temporal  pattern  discrimination  were  conducted.  Listener  performance  in  these 
experiments  was  evaluated  using  a  mathematical  model  of  temporal  pattern  discrimination 
called  the  Pattern  Correlation  Model.  Analyses  of  these  expenments  allow  specification 
of  the  temporal  pattern  discrimination  mechanisms  employed  by  the  human  auditory 
system. 

Our  experiments  tested  how  listeners  discriminate  between  arrhythmic,  tonal 
sequences  that  were  approximately  one  half-second  in  duration.  According  to  the  Pattern 
Correlation  Model,  the  listener  extracts  a  list  of  tone  marker  interonset  times  from  each 
pattern,  and  then  computes  the  correlation  between  the  pattern  of  time  intervals  marked 
by  the  tones  in  each  sequence;  other  information  about  the  input  waveforms  (such  as 
absolute  timing  or  spectra)  is  discarded.  The  experiments  tested  how  listener  performance 
depends  on  basic  parameters  of  the  task,  such  as  sequence  correlation,  and  number, 
duration,  and  variability  of  pattern  elements.  Listener  performance  was  consistent  with 
the  predictions  of  the  Pattern  Correlation  model,  but  was  limited  by  an  internal  time  jitter 
or  noise  that  was  a  function  of  the  average  intermarker  interval. 

A  second  study  evaluated  the  human  listener’s  ability  to  discriminate  between 
word-length  tonal  sequences  that  were  subjected  to  uniform  temporal  transformations, 
such  as  time  compression  and  expansion.  One  of  the  most  intriguing  features  of  temporal 
pattern  perception  is  the  ability  to  recognize  patterns  as  similar,  despite  such  compression 
or  e;q)ansion  manipulations.  Examples  of  such  time  normalization  abound  in  speech  and 
music  perception  and  we  are  normdly  unaware  of  such  temporal  changes,  even  when  they 
occur  during  relatively  brief  stimuli,  such  as  words.  These  experiments  tested  how  well  the 
Pattern  Correlation  model  could  predict  the  effects  of  time  compression  and  expansion  on 
listener  performance.  The  model  proved  useful  in  describing  performance  in  a  variety  of 
different  conditions  that  employed  multiplicative  and  additive  time  transformations. 
Listener  performance  dropped  when  one  of  the  sequences  was  compressed  or  expanded  in 
time.  In  order  for  the  model  to  describe  performance,  it  was  necessary  to  postulate  an 
additional,  internal  noise  component  that  was  proportional  to  the  ma^tude  of  the 
difference  between  the  sequence  transformations. 

A  third  study  evaluated  the  possibility  that  different  pattern  comparison 
mechanisms  operate  in  different  task  conditions.  The  experiments  evaluated 
discrimination  when  the  two  patterns  began  at  delayed  starting  times.  The  patterns  were 
presented  at  different  frequencies  and  to  different  ears,  and  were  subjected  to 
multiplicative  compressions  and  expansions.  Listeners  performed  well  even  when  the 
patterns  contained  tones  of  different  frequency  and  in  spite  of  the  patterns  being 
presented  to  separate  earphone  channels.  Performance  was  good  when  the  sequences 
were  presented  either  (near)  simultaneously  or  at  relatively  long  time  delays.  When  the 
time  between  pattern  onsets  was  less  than  10-ms,  discrimination  was  veiy  sensitive  to  the 
expansion  or  compression  manipulation,  indicating  that  discrimination  in  this  region  was 
based  on  the  process  of  waveform  correlation.  At  longer  time  separations,  performance 
was  relativety^insensitive  to  such  transformations,  consistent  with  the  Pattern  Correlation 
hypothesis.  Thus  the  results  support  a  two-phase  mechanism:  when  the  sequence  delay  is 
less  than  20-ms,  the  binaural  waveform  correlator  is  the  active  mechanism;  when  the 
sequence  delay  is  greater  than  20-ms,  the  pattern  correlator  is  the  active  mechanism. 
Moreover,  the  efficiency  of  the  pattern  correlation  mechanism  is  very  poor  when  the 
sequences  overlap  in  time.  It  appears  that  the  sequential  presentation  of  stimulus  patterns 
is  a  requirement  for  the  pattern  correlator  to  function. 
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We  are  now  conducting  experiments  on  the  role  of  rhythm  in  the  discrimination 
and  perception  of  complex  auditory  patterns.  Two  emerimental  paradigms  are  being 
employed.  In  one,  we  are  investigating  the  possible  advantages  of  cyclic  repetitions  in  the 
temporal  patterns  to  be  discriminated.  Because  the  presence  of  cyclic  repetition  or 
pattern  correlation  (i.e.  rhythm)  reduces  the  amount  of  (independent)  statistical 
information  in  the  stimuli  to  be  compared,  this  paradigm  offers  a  way  to  extend  the 
Pattern  Correlation  Model.  A  second  experimental  paradigm  attempts  to  assess  directly 
the  listener’s  ability  to  discriminate  rhythmicity  as  measured  by  the  pattern  correlation. 
This  e^eriment  oners  the  possibility  that  the  Pattern  Correlation  Model  may  be  useful  for 
modeling  the  perception  of  rhythm. 

In  addition  to  auditory  experiments  on  temporal  pattern  processing,  we  completed 
two  studies  of  visual  pattern  processing.  One  continuing  study  employs  a  weighting 
analysis  derived  from  signal  detection  theory;  the  goal  is  to  specify  how  the  observer’s 
processing  of  information  from  different  spatial  positions,  varies  as  a  function  of  the  stage 
of  information  processing  and  the  time  (and  other)  stress  on  the  observer. 

Finally,  we  have  begun  the  development  and  computer  simulation  of  group  signal 
detection.  Group  signal  detection  systems  are  composed  of  m  detectors,  who  must  work 
together  to  detect  a  weak  signal  in  noise.  These  analyses  have  already  resulted  in  some 
very  interesting  comparisons  of  the  relative  efficiency  of  different  detection  schemes, 
including  common  systems  such  as  the  traditional  jury. 
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I.  OBJECTIVES  AND  STATUS  OF  THE  RESEARCH  EFFORT 


1.  Discrimination  of  arrhythmic  tonal  sequences:  Effect  of  temporal  transformations. 
(Sorkin  and  Montgomery). 

A  basic  aspect  of  our  normal  perception  of  speech  and  music,  is  the  ability  to 
discriminate  and  categorize  temporal  patterns.  We  have  been  studying  the  discrimination 
of  brief  tonal  sequences.  The  observer  is  presented  with  two  sequences  of  tones  and  asked 
to  report  whether  or  not  the  two  temporal  patterns  are  the  same-ignoring  any  other 
differences  between  the  two  stimuli,  such  as  frequency.  On  half  the  trials,  the  patterns  are 
the  same,  and  on  half  they  are  different.  For  most  of  our  experiments,  the  sequences  were 
composed  of  8, 30-ms  duration,  1000-Hz  tones,  and  the  mean  intertone  interval  was  set 
between  20  and  100-ms.  TTie  mean  duration  of  a  typical  sequence  was  600-ms  and  the 
time  separation  between  the  pair  of  sequences  was  750-ms. 

Subjects  perform  this  task  very  well.  The  important  variable  controlling  task 
difficulty  is  the  correlation,  between  the  two  series  of  tone  interonset  times.  On 
SAME  trials,  p  is  set  to  1.0  and  on  DIFFERENT  trials,  is  set  to  a  constant  value  less 
than  1.0,  depenmng  on  the  condition  of  interest.  The  task  is  easiest  when,  on 
DIFFERENT  TRIALS,  p^^  is  set  to  zero,  and  becomes  more  difficult  as  p^^  approaches 
one.  The  optimal  way  to  perform  this  task  is  to  estimate  p^^  from  the  sequences  of 
interonset  times  observed  on  a  trial.  We  know  how  that  statistic  is  distributed,  and  so  we 
can  compute  how  d’  should  depend  on  rho  and  the  number  of  tones  in  the  sequence.  We 
assume  that  there  is  an  internal  noise  or  jitter  on  the  subject’s  estimate  of  the  times,  and 
we  factor  that  into  our  prediction  of  subject  performance.  For  our  previously  reported 
data,  this  jitter  was  approximately  15-nis. 

We  tested  sequence  discrimination  when  the  second  sequence  was  time  expanded 
or  compressed  by  a  multiplicative  or  additive  constant.  A  mechanism  that  extracts  the 
times  and  then  computes  the  correlation  between  the  two  lists  of  intertone  times,  e.g.  a 
temporal  pattern  correlator,  will  be  relatively  insensitive  to  such  time  transformations. 
(Imagine  that  you  multiply  each  element  in  me  second  of  two  lists  of  numbers,  by  a 
constant.  The  correlation  should  not  be  affected  by  that  multiplication.)  We  found  that 
performance  was  good  under  the  transformations,  but  was  affected  by  the  magnitude  of 
the  transformation  to  the  second  sequence.  A  satisfactory  model  of  performance  used  two 
sources  of  internal  noise:  a  fixed  component  of  internal  noise  and  a  component  dependent 
on  the  magnitude  of  the  transformation  difference  between  the  sequences. 


2.  Discrimination  of  arrhythmic  tonal  sequences:  Effect  of  delay  in  onset  of  second 
sequence.  (Sorkin  and  Montgomery). 

These  experiments  evaluated  temporal  pattern  discrimination  as  a  function  of  me 
time  delay  between  the  patterns.  Listeners  decided  whether  two  arhythmic  tonal 
sequences  had  the  same  or  different  temporal  patterns.  The  patterns  were  determined  by 
the  sequence  of  successive  time  intervals  between  tones.  According  to  the  Pattern 
Correlation  Model,  listeners  discriminate  between  arrhythmic  tonal  sequences  by  (a) 
extracting  information  about  the  time  intervals  between  the  tones  and  (b)  computing  me 
correlation  between  the  serial  pattern  of  time  intervals  in  each  sequence.  We  examined 
the  effect  of  varying  the  time  interval  between  onsets  of  the  pair  or  sequences;  we  wished 
to  evaluate  delays  of  from  a  few  ms-up  to  the  relative  long  delays  of  our  previous 
experiments.  The  sequences  were  presented  at  different  frequencies  and  separately  to  the 
two  ears. 
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Performance  was  quite  good  when  the  sequence  onsets  began  within  1-ms  and 
decreased  as  the  delay  between  onsets  exceeded  20-ms,  approaching  a  minimum  at  300-ms 
(when  the  sequences  overlap  by  50%)  and  then  increasing  again.  TTie  average  duration  of 
each  sequence  was  about  bOO-ms.  The  moderately  good  performance  at  long  delays  is 
consistent  with  our  previous  experiments  and  with  the  idea  that  the  task  is  performed  by 
estimating  the  temporal  correlation  between  the  time  patterns  defined  by  the  intertone 
time  intervals. 

We  would  expect  very  good  performance  at  very  short  onset  delays;  because  under 
these  conditions,  the  binaural  system  can  make  very  precise  determinations  of  differences 
between  stimuli  presented  to  the  two  ears.  Other  investigators  have  reported  that  binaural 
comparisons  may  be  performed  when  the  signals  are  presented  at  high  frequencies  (e.g. 
above  2,000  Hz),  and  at  different  frequencies  in  the  two  earphone  chaimels.  A  delay  in 
the  second  signal  of  more  than  about  15  milliseconds  would  be  expected  to  exceed  the 
limits  of  the  ftnaural  system.  We  would  expect  that  a  mechanism  that  simply  correlated 
the  two  input  waveforms  would  be  very  sensitive  to  the  time  expansion  manipulation, 
because  temporal  misalignments  that  occurred  early  in  the  sequence  would  produce  even 
greater  decorrelations  at  later  times. 

In  order  to  examine  the  interaction  between  these  manipulations,  we  combined  the 
two  manipulations  of  temporal  expansion  and  intersequence  delay.  That  is,  we  added  a 
condition  in  which  all  time  intervis  in  the  second  sequence  were  expanded  by  a  small 
uniform  factor  between  0.8  and  1.2.  Over  trials,  we  multiplied  all  tone  durations  and  gaps 
in  the  second  sequence  by  either  0.8, 0.9, 1.0, 1.1  or  1.2,  chosen  randomly  over  trials.  The 
results  are  shown  in  the  figure:  The  solid  curves  are  the  conditions  with  no  time 
transformations.  The  dashed  curves  show  the  random  time  transformations.  In  the  latter 
case,  performance  at  very  short  delays  dropped  almost  to  zero.  It  is  tempting  to  conclude 
that  the  mechanism  operating  at  short  onset  delays  is  a  waveform  correlation  process;  and 
the  mechanism  operating  at  long  delays,  is  a  temporal  pattern  correlation  process. 

Why  is  neither  mechanism  effective  at  intermediate  delays?  These  delays  are 
clearly  too  long  for  the  binaural  system,  so  the  temporal  pattern  correlator  is  left  to  do  the 
job-and  apparently  it  cannot.  In  order  for  the  relevant  time  information  to  be  extracted 
from  the  stiinuli,  the  sequences  may  need  to  be  presented  to  the  system-one  sequence  at  a 
time-that  is,  in  a  "single-channel"  mode.  It  may  be  that  this  mode  of  processing  requires 
the  observer’s  attention  to  the  individual  stimuli  to  be  compared. 
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Performance  (d’)  as  a  function  of  intersequence  interval,  under  two  temporal 
transformation  conditions:  no  transformation  (solid  lines),  and  random-transformations 
(dashed  lines).  The  panels  show  data  for  three  values  of  mtersequence  correlation. 
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3.  Effects  of  rhythm  on  temporal  pattern  discrimination  (Sorkin  and  Sadralodabai). 

This  ongoing  project  assesses  the  effects  of  the  rhythmic  properties  of  the  temporal 
patterns  on  the  listener’s  ability  to  discriminate  between  patterns.  The  success  of  the 
pattern  correlation  model  suggested  that  its  ability  to  describe  a  listener’s  temporal 
pattern  discrimination  behavior  may  be  generalizable  to  "rhythmic",  e.g.  repeating,  non- 
random  patterns.  In  this  experiment,  we  use  the  pattern  correlation  model  to  characterize 
the  rhythmicity  of  a  sequence;  that  is,  the  temporal  correlation  between  repeated 
sequences  of  length  k,  is  taken  as  a  first  approximation  to  a  measure  of  stimulus 
rhjhhmicity.  The  hypothesis  under  study  m  this  e^eriment  is  whether  two  rhythmic  (or 
partially  rhythmic)  patterns  are  more  easily  discriminated  than  are  non-rhythmic  patterns. 

One  difficulty  in  performing  this  experiment  is  to  insure  that  any  difference  in  the 
rhythmicity  of  the  two  sequences  to  be  discriminated  does  not  itself  signal  a  difference 
between  the  sequence  patterns.  We  have  designed  and  are  currently  running,  an 
experiment  in  which  these  variables  are  controlled.  If  pattern  rhythmicity  is  found  to 
affect  temporal  pattern  discrimination,  we  will  attempt  to  model  the  process  and  define 
what  factors,  such  as  memory,  mediate  those  effects. 

4.  Discrimination  of  Pattern  Rhythmicity:  Extended  Temporal  Correlation  Model  (Sorkin 
and  Sadralodabai). 

The  initial  results  of  the  rhythmic  pattern  discrimination  experiment  have  led  us  to 
study  the  listener’s  ability  to  discriminate  the  extent  of  rhythmicity  in  a  temporal  pattern. 
Using  the  temporal  pattern  correlation  model  to  provide  an  objective  measure  of  pattern 
rhymicity,  we  have  begun  experiments  assessing  a  listener’s  ability  to  discriminate  this 
aspect  of  a  temporal  stimulus.  Essentially,  the  listener  is  provided  with  two  temporal 
patterns  (each  is  a  sequence  composed  ot  3  repetitions  of  4  intertone  intervals)  and  must 
discriminate  which  pattern  is  more  rhythmic.  We  will  employ  the  pattern  correlation 
model,  augmented  by  assumptions  about  an  additional  source  of  internal  jitter,  to  model 
this  discrimination  process. 

5.  Analysis  of  Group  Detection  Systems  (Sorkin  and  Crandall). 

In  this  study,  we  are  using  the  theory  of  signal  detectability  to  predict  how  the 
performance  of  grouped  detection  systems  depends  on  the  size  of  the  group,  the  group 
decision  rule,  and  the  characteristics  of  the  group  members.  These  systems  are  assumed  to 
have  m  individual  detectors,  each  with  a  specified  detection  sensitiviw  and  response 
criterion.  Four  types  of  groups  are  considered:  (1)  the  Ideal  Group,  (b)  the  Standard  Jury, 
(c)  the  Limited  Interaction  (Delphi)  Jury,  and  (d)  the  Single  Ballot  Group.  Decisions  of 
the  Ideal  Group  are  determined  by  wei^ting  and  summing  the  likelihood-ratio 
observations  of  the  individual  detectors;  this  strategy  defines  the  upper  bound  of 
performance  achievable  by  any  group  detection  system.  Lower  performance  is  obtained 
when  the  group  decisions  are  based  on  the  binary  outputs  of  the  individual  detectors.  The 
Standard  Jury  is  assumed  to  function  as  follows:  After  the  observation  is  made,  each 
detector  makes  an  initial  decision  and  then  recomputes  its  response  criterion  as  a  function 
of  the  initial  decision,  sensitivity,  and  criterion,  of  the  other  detectors.  This  cycle  repeats 
until  unanimity  (or  other  majority  decision  rule)  is  reached  or  time  runs  out. 

We  have  performed  computer  simulations  of  these  group  detection  systems  for  a 
variety  of  conditions,  group  member  parameters,  and  majority  voting  rules.  The  Standard 
Jury  exhibits  the  best  performance  of  the  non-optimal  (binary  combination)  groups.  We 
shall  consider  the  implications  of  the  analysis  for  the  design  of  group  detection  systems. 
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6.  Inteffation  of  information  from  multiple  element  displays  (Sorkin,  Mabry,  Weldon,  and 
Elvers), 

This  experiment  examined  the  processing  of  information  from  multiple  element 
visual  displays,  using  techniques  derived  from  the  theory  of  signal  detectability.  The 
method  allows  one  to  specify  how  observers  integrate  information  from  individual 
elements  of  a  display.  The  experiment  tested  numerical  and  graphical  displays  having 
different  display  sizes,  durations,  and  arrangements  of  elements.  Observer  performance 
increased  with  the  number  of  display  elements  (m),  but  at  less  than  the  ideal  rate. 
Observer  performance  was  consistent  with  a  model  of  information  integration  constrained 
by  internal  noise.  Linear  arrays  of  elements  resulted  in  better  performance  than  did 
square  arrays.  Graphically  coded  elements  resulted  in  better  performance  than  did 
numerically  coded  elements.  Observer  decision  weighting  of  element  information  from 
graphical  displays  was  approximately  um'form  across  spatial  positions,  but  the  weighting  of 
information  from  numerical  displays  was  concentrated  on  elements  near  the  fixation  point. 

7.  Information  integration  under  processing  limitations  (Montgomery). 

Three  experiments  were  performed  to  determine  the  effects  of  time  stress  on  the 
selection  and  use  of  visually  displayed  information  in  a  2*alternative-forced-choice 
decision  task.  Observer  time  stress  was  varied  by  altering  the  stimulus  duration, 
complexi^,  and  time  to  respond.  The  stimulus  consisted  of  a  horizontal  array  of  nine, 
three-digit  numbers.  Each  number  was  independently  sampled  from  either  a  signal  or 
noise  probability  distribution,  depending  on  the  type  of  trial.  Observers  had  to  decide 
whether  the  display  was  generated  fi'om  the  signal  or  noise  distribution.  A  technique 
developed  by  Bruce  Berg,  provided  the  means  for  characterizing  changes  in  observer 
performance  from  ideal  performance  as  (1)  changes  in  the  relative  decision  weights  for  the 
different  spatial  positions  or  (2)  changes  m  the  observer’s  internal  noise. 

In  the  first  two  experiments  time  limitations  were  imposed  during  sensory 
processing  and  response  selection.  Limitations  in  capacity  or  bandwidth  at  the  sensory 
processing  stage  were  expected  to  reduce  the  number  of  ^owable  fixations,  leading  to  a 
reduction  in  the  information  available  from  different  spatial  positions.  Limitations 
imposed  later  in  processing ,  closer  to  response  selection  and  execution,  were  expected  to 
force  observers  to  use  whatever  information  was  available  for  a  rapid  decision, 
interpretable  as  a  non-spatially  specific  loss  in  performance  due  to  internal  noise. 

Evidence  from  both  experiments  supported  the  early  stage  effect:  neither  limiting  the 
answer  duration  nor  forcing  and  immediate  response  produced  significant  effects  on 
performance.  In  the  third  experiment,  stimulus  complexity  was  manipulated  by  altering 
the  mean  and  variance  of  the  two  signal  distributions.  Changes  in  these  parameters  did 
not  produce  significant  changes  in  performance. 

II.  PERSONNEL  ASSOCIATED  WITH  THE  RESEARCH  PROJECT 

Crandall,  Christian  S.,  Visiting  Assistant  Professor  of  Psychology,  University  of 
Florida.  Professor  Crandall  has  been  assisting  with  the  group  detection  study. 

Montgomery,  D.  A.  (Widman).  Graduate  Student,  Department  of  Psychology, 
University  of  Florida.  Ms.  Montgomery  has  been  on  the  project  since  May,  1989. 

Sadralodabai,  Toktam.  Graduate  Student,  Department  of  Psychology,  University  of 
Florida.  Ms.  Sadralodabai  joined  the  project  in  August,  1991, 

Sorkin,  R.  D,  Principal  Investigator,  Professor  of  Psychology,  University  of  Florida. 
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selectivity.  Paper  presented  at  the  35th  Annual  Meeting  of  the  Human  Factors  Society, 

San  Francisco,  California,  Sept.  1991. 

Sorkin,  R.  D.  and  Widman,  D.  A.  Discrimination  uf  arrhythmic  tonal  sequences; 
Effect  of  temporal  transformations.  (Paper  presented  at  120th  Meeting  of  the  Acoustical 
Society  of  America),  Journal  of  lh£  Acoustical  Society  of  America.  1990,  S147, 
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